Skip to content

Hot-reload .emanoteignore mid-session (closes #739)#741

Draft
srid wants to merge 19 commits into
masterfrom
hot-reload-emanoteignore
Draft

Hot-reload .emanoteignore mid-session (closes #739)#741
srid wants to merge 19 commits into
masterfrom
hot-reload-emanoteignore

Conversation

@srid

@srid srid commented May 17, 2026

Copy link
Copy Markdown
Owner

emanote run now picks up edits to a layer's .emanoteignore without a restart. Adding a pattern evicts matching notes and static files from the model and the sidebar; removing a pattern brings the previously-hidden ones back. Closes #739.

The seam the issue described — unionmount accepts ignore patterns as a static Map source [FilePattern] — is sidestepped by moving per-layer filtering out of unionmount entirely. unionmount keeps only the universal patterns (dotfile-dir contents, vim backups, the reserved -/); per-layer .emanoteignore content lives in a TVar that .emanoteignore change events update in place. The streaming handler reads that TVar on every event and the dedicated reload path walks _modelNotes + _modelStaticFiles whenever the pattern set actually moves.

Flow

fsnotify event
      │
      ▼
unionmount Change ──▶ handle (splits IgnoreFile from rest)
                          │
                          ├── R.IgnoreFile  ─▶  handleIgnoreFileChanges
                          │                       ├── reload patterns from disk
                          │                       ├── evictNewlyIgnored  (notes + static files)
                          │                       └── reEmitModified     (ledger walk)
                          │
                          └── everything else ─▶ applyFiltered ─▶ classifyOverlays
                                                                    ├── Kept     → patchModel as-is
                                                                    ├── Partial  → patchModel with survivors + record
                                                                    └── Dropped  → Delete + record

Emanote.Route.Ext gains a new IgnoreFile constructor so unionmount surfaces .emanoteignore events through the same tagged pipeline as .md/.lua/.yaml. The R.IgnoreFile branch of patchModel is a no-op — by the time an event reaches it, handleIgnoreFileChanges has already reshaped the model upstream.

Coverage

Two new @live @hot-reload Cucumber scenarios in tests/features/emanoteignore.feature cover both directions:

Scenario Setup Action Expectation
Removing a pattern re-includes the matching note Baseline .emanoteignore excludes secret-ignored.md Rewrite file without that line URL /secret-ignored.html contains marker within 10s
Adding a pattern hides the matching note Empty .emanoteignore Rewrite to add secret-ignored.md URL stops containing marker within 10s

Behavior change

The previous universal **/.*/** pattern silently matched root-level standalone dotfiles too (because filepattern's ** matches the empty path). Tightening it to **/.*/**/* is what lets .emanoteignore reach the hot-reload pipeline; the side effect is that root-level .gitignore, .envrc, .editorconfig, etc. — which the old pattern was incidentally swallowing — are now exposed as static assets unless the user lists them in their own .emanoteignore. Dotfile directories like .git/ and .vscode/ are still filtered, just by the tighter pattern.

Notable limitations (documented in docs/guide/emanoteignore.md)

YAML data files (the metadata cascade) and Heist .tpl templates don't carry per-layer source metadata in the model yet, so adding a pattern matching one mid-session leaves the resolved cascade / template loaded until restart. Removing such a pattern is unaffected because those files were never filtered out in the first place.

Try it locally

nix run github:srid/emanote/hot-reload-emanoteignore -- -L ./my-notebook run

Generated by /do on Claude Code (model claude-opus-4-7).

srid added 19 commits May 17, 2026 14:27
Each layer's .emanoteignore was read once at process start; editing
the file during `emanote run` required restarting the binary to take
effect. The seam noted in Source/Dynamic.hs is removed by moving
per-layer ignore filtering out of unionmount (which only sees the
fixed universal patterns now) and into the streaming handler in
Emanote.Source.Dynamic: per-layer patterns live in a TVar that the
fsnotify pipeline updates in place when the .emanoteignore file
itself changes, the handler walks the model to evict newly-ignored
notes, and walks a tracked set of previously-suppressed events to
re-include files whose patterns were removed.

New 'IgnoreFile' FileType claims .emanoteignore as a first-class
source so unionmount delivers its events through the same pipeline
as other source files. The static-file index branch in
Source.Patch skips it; the patchModel branch is a no-op because
Source.Dynamic has already applied the effect by the time patchModel
sees the event.

Covered by two new live @Hot-Reload cucumber scenarios in
tests/features/emanoteignore.feature exercising both directions
(pattern added → note hidden, pattern removed → note re-appears).
Sub-tree .emanoteignore events (e.g. sub/.emanoteignore) carry the
same R.IgnoreFile tag as the layer-root file but configure no
patterns. Skip the disk reload when a batch contains only sub-tree
events; previously the predicate was exported but had no caller.

Closes hickey finding #5 / lowy finding #2.
…noreFileChanges

LuaFilter hot-reload is implemented inside Patch.patchModel; the
.emanoteignore hot-reload runs *upstream* of patchModel, inside
Source.Dynamic. A reader could miss the asymmetry without a pointer
at each end — add explicit cross-references to keep the structure
discoverable.

Closes lowy finding #4.
Matches the codebase convention (ModelT, Note, StaticFile, EmanoteConfig)
of accessing internal record state through optics rather than raw
underscore-prefix selectors.

Closes lowy finding #3.
applyFiltered and reEmitModified both ran their own NE.filter +
length comparison + nonEmpty branch to decide whether to dispatch
Refresh, Refresh-with-trimmed-overlays, or Delete. Lift the pure
decision into 'Emanote.Source.Ignore.classifyOverlays' returning an
'OverlayOutcome' so both call sites read as: classify → branch →
dispatch. The bookkeeping (recordModified/forgetModified) stays at
the call site where it belongs.

Closes hickey finding #4.
…lice

Previously _isPatterns and _isModifiedEvents were two TVars whose
invariants (entry absent from set ↔ visible in model; entry present
↔ suppressed) were held by hand at three sites. Combine them into a
single 'IgnoreSlice' record behind one TVar and route every read
through 'snapshotSlice' / every write through 'updateSlice'. The
single-transaction property gives a future invariant linking the two
fields one place to live, and 'recordModified' / 'forgetModified' now
delegate to the slice helper instead of poking a raw TVar.

Closes hickey finding #3.
applyFiltered, applyOne, handleIgnoreFileChanges, evictNewlyIgnored,
and reEmitModified each threaded the same four invariants positionally
(layers, noteFn, storkIndex, ignoreState). Bundle them into a single
HandlerCtx so a new helper that needs (say) storkIndex alongside the
ignore state doesn't have to grow another positional argument.

Closes hickey finding #6.
evictNewlyIgnored only walked _modelNotes, leaving static files
(images, PDFs, source-code embeds, ...) serving when their path
matched a freshly-added pattern. Add _staticFileSource :: Maybe
(Loc, FilePath) to StaticFile, populate it from the top overlay in
Patch.insertStaticFile, and walk _modelStaticFiles alongside notes
in evictNewlyIgnored.

The hot-reload coverage for YAML data and Heist templates remains a
known limitation — neither carries per-layer source metadata in the
model today. Documented in docs/guide/emanoteignore.md.

Closes hickey finding #2 / lowy finding #1.
…o their guides

The hot-reload caveat for YAML data and Heist templates now points at
[[yaml-config]] and [[html-template]] so a reader following the
limitation has the surrounding context one click away.
evictNewlyIgnored unconditionally overwrote any existing ledger entry
with a single-element ((loc, lfp) :| []) overlay drawn from
_noteSource / _staticFileSource. If a previous OverlayPartial had
stashed the full multi-layer overlay (e.g. [(Layer1, foo.md),
(Layer2, foo.md)]), that history would be silently lost — a
subsequent pattern relaxation would resurrect the file from the
wrong layer in multi-layer setups.

Add 'recordModifiedIfAbsent' and route both note and static-file
eviction through it. The original 'recordModified' (overwrite-wins)
stays the right helper for 'applyFiltered', where the new event from
unionmount carries the freshest overlay.
…leType

The handwritten isMdRoute case-split duplicates the 'HasExt' class's
'fileType' method, which already resolves to the correct
'R.LMLType R.Md' / 'R.LMLType R.Org' per its instance. Route via
'withLmlRoute' so the type-class machinery handles the choice.
snapshotSlice / updateSlice / changeEntries / applyOne / mountedPath
each had a one-line Haddock that restated the identifier name. Per
the codebase convention (default to no comments; keep only non-obvious
why), delete them. The two record-helpers (recordModified vs
recordModifiedIfAbsent) keep shortened comments naming the policy
choice — that's the non-obvious distinction.
…e with pointer

The 14-line comment above the R.IgnoreFile branch in Patch.hs repeated
prose that already lives in the haddock for handleIgnoreFileChanges in
Dynamic.hs. Replace with a 3-line cross-reference; Dynamic.hs is the
single source of truth for the LuaFilter-vs-IgnoreFile asymmetry.
…erns empty

Every fsnotify event during a live-reload session runs through
classifyOverlays. For the common case where no layer has any
per-layer patterns, the NE.filter walk + per-overlay Map lookups are
all no-op work. Add a fast path that returns OverlayKept directly
when the pattern map is empty.
…patterns changed

evictNewlyIgnored and reEmitModified previously walked the full
_modelNotes and _modelStaticFiles on every .emanoteignore edit. For a
10k-note notebook each save was O(N), even though a typical edit
touches only one layer's pattern set.

Diff old vs new patterns per Loc once, then filter every walk by Set
membership before applying the heavier isLayerPathIgnored /
classifyOverlays predicates. Correctness: if a layer's patterns
didn't move, no file in that layer can have flipped sides.
Three step definitions (and one new one from this PR) repeated the
same path.join + mkdirSync({recursive: true}) + writeFileSync trio.
Lift it into support/fixture.ts next to stagedFixtureDir; both
step files reuse it. A future step that writes a new fixture file
gets the right ceremony for free.
…steps

Three step definitions ('URL X contains Y within Ns', 'URL X stops
containing Y within Ns', 'I wait for X to contain Y') used the same
request-inspect-sleep-retry loop. Lift it into tests/support/poll.ts
behind a typed predicate + onTimeout pair so each step shrinks to a
4-line wrapper; the page-DOM 'article body contains … within Ns'
step stays put since it uses Playwright's waitForFunction.
filepattern's '**' matches the empty path, so the master pattern
'**/.*/**' silently matched a root-level dotfile like '.emanoteignore'
(via ** = "", .* = ".emanoteignore", trailing ** = ""). The previous
ignorePatterns list passed this to unionmount as a per-source ignore;
unionmount's fsnotify watcher then filtered every .emanoteignore
modify event before it could reach handleIgnoreFileChanges, breaking
the entire hot-reload pipeline this PR was meant to deliver.

Tighten the pattern to '**/.*/**/*' so the trailing component is
required. Dotfile-directory CONTENTS (.git/HEAD, .vscode/settings.json,
.git/objects/00/abc) still match; root-level standalone dotfiles
(.emanoteignore, .gitignore, .envrc, …) no longer do.

Behavior change: root-level dotfiles that the universal pattern was
incidentally filtering — .gitignore, .envrc, .editorconfig, etc. —
are now exposed as AnyExt static files unless the user adds them to
their own .emanoteignore. Most users have nothing sensitive in those
files; users who do can list them in .emanoteignore.

Verified manually: with this fix the two new hot-reload scenarios in
tests/features/emanoteignore.feature flip the matching note's
visibility in well under the 10-second polling budget.
The hot-reload entry now flags that the universal pattern change to
let .emanoteignore through also stops swallowing other root-level
standalone dotfiles (.gitignore, .envrc, .editorconfig, …). Dotfile
*directories* (.git/, .vscode/) are unchanged.
@srid

srid commented May 17, 2026

Copy link
Copy Markdown
Owner Author

Hickey/Lowy Analysis

# Lens Finding Disposition
1 Hickey **/.*/** claim — silently matches root .emanoteignore No-op (empirically wrong: filepattern test verified True/True/True for the pattern matching root + nested cases. The static "the .emanoteignore file itself is not served" scenario on master also depends on this match working at root.)
2 Hickey evictNewlyIgnored skips static files Fixed in this PR
3 Hickey _isModifiedEvents parallel mirror with discipline-only invariants Fixed in this PR
4 Hickey applyFiltered complects filter + bookkeeping + dispatch; predicate duplicated Fixed in this PR
5 Hickey isLayerRootIgnoreFile exported but unused Fixed in this PR
6 Hickey Session-stable params (layers, noteFn, storkIndex, IgnoreState) threaded positionally Fixed in this PR
1 Lowy evictNewlyIgnored is note-only — scope asymmetric to applyFiltered Fixed in this PR (subsumed by Hickey #2 + docs caveat for YAML/templates)
2 Lowy isLayerRootIgnoreFile dead exported interface Fixed in this PR (same finding as Hickey #5)
3 Lowy IgnoreState / ModifiedEvent raw accessors without makeLenses Fixed in this PR
4 Lowy Asymmetric Patch.IgnoreFile vs Dynamic.handleIgnoreFileChanges interception points undocumented Fixed in this PR

Hickey rationale

The diff initially folded both filtering decisions and ledger bookkeeping into single match arms of applyFiltered — three Refresh cases each pairing a pure outcome (kept / partial / dropped) with a side-effecting record-or-forget plus a dispatch. Lifting the pure decision into Ignore.classifyOverlays (returning an OverlayOutcome ADT) lets applyFiltered and reEmitModified read as classify → bookkeep → dispatch instead of triple-interleaved per-arm code; the same predicate now lives in one place.

The two TVars (_isPatterns + _isModifiedEvents) were folded into one IgnoreSlice behind a single TVar. The discipline that every recordModified should pair with a matching forgetModified is enforced by routing every read through snapshotSlice and every write through updateSlice — both helpers act on the single slice atomically, so a future invariant linking the two fields has one place to live.

evictNewlyIgnored originally only walked _modelNotes, leaving static files (PDFs, images, source-code embeds) reachable even after a pattern was added that should hide them. Adding _staticFileSource :: Maybe (Loc, FilePath) to StaticFile and walking _modelStaticFiles alongside notes closes the gap. YAML data and Heist .tpl templates carry no source metadata yet — that limitation is documented in the user-facing docs page rather than fixed structurally; both volume and scope made the model-layer refactor disproportionate to the PR's mandate.

Session-stable handler parameters (layers, noteFn, storkIndex, IgnoreState) were threaded positionally through five functions; bundled into a HandlerCtx record so a new helper that needs (say) storkIndex alongside ignoreState doesn't grow another positional argument.

Lowy rationale

The volatility being encapsulated is "what counts as a layer-root configuration file." Master left it overloaded with "what kinds of source files unionmount routes through the typed pipeline".emanoteignore was filtered universally, with no first-class type. Adding R.IgnoreFile to FileType SourceExt makes that volatility visible: Pattern.filePatterns claims the file (with IgnoreFile listed before AnyExt so it wins tagging), Patch.indexesAsStaticFile excludes it from the static-file route, and Source.Dynamic.handleIgnoreFileChanges owns the reshape semantics. A future change to "what counts as a layer-root config file" lands in one place.

The _sliceModified ledger encapsulates a volatility orthogonal to the model: the model tracks resolved content, the ledger tracks the filter decisions that produced that content. Re-walking the filesystem on every pattern change would couple the orchestration to filesystem I/O — keeping the modified-event mirror is the right call for this volatility split.

Emanote.Source.Ignore (file format + pattern matching) and Emanote.Source.Dynamic (mid-session orchestration) are the right two modules for the two volatilities. isLayerRootIgnoreFile ended up in Ignore as a name predicate (pure semantics) and is consumed by Dynamic for event routing — clean boundary. The asymmetric interception points between Patch.patchModel (LuaFilter hot-reload) and Dynamic.handleIgnoreFileChanges (.emanoteignore hot-reload) are structurally justified — different volatilities, different lifecycles — but easy to miss without explicit cross-reference notes, which the diff now adds at both ends.

@srid

srid commented May 17, 2026

Copy link
Copy Markdown
Owner Author

Evidence

Three WebSocket-morphed states captured in a single browser tab at /secret-ignored — no page reload between states; the sidebar and content update purely from fsnotify → Ema-WS.

State 1 — Baseline. secret-ignored.md is listed in .emanoteignore. The route renders the "Missing link" error page and the note is absent from the sidebar.

Baseline: note excluded, Missing link error

State 2 — Pattern removed. The secret-ignored.md line is deleted from .emanoteignore on disk. The page morphs in place: the note body appears (marker text EMANOTEIGNORE_REGRESSION_FILE is visible) and "Should not appear" is listed in the sidebar.

Pattern removed: note body visible in sidebar

State 3 — Pattern re-added. The line is written back to .emanoteignore. The page morphs back to "Missing link" and the entry disappears from the sidebar — without restarting the server or reloading the tab.

Pattern re-added: note hidden again

@srid

srid commented May 17, 2026

Copy link
Copy Markdown
Owner Author

/do results

Step Status Duration Verification
sync 0s forge=github, branch=hot-reload-emanoteignore
research 12m 10s design settled: emanote-only TVar+ledger approach
branch 5s feature branch hot-reload-emanoteignore
implement 15m 6s IgnoreFile FileType; classifyOverlays; TVar-driven hot-reload; 2 cucumber scenarios
check 14s cabal build all clean
docs 1m 17s CHANGELOG bullet + emanoteignore.md Hot reload section
fmt 49s hlint + fourmolu + cabal-fmt + nixpkgs-fmt all green
commit 24s primary feature commit pushed
hickey+lowy 34m 50s 10 findings: 1 No-op (empirically incorrect), 7 fix commits, 2 cross-lens dups
police 28m 9s 1 rule fix + 1 fact-check + 7 elegance fixes (9 commits)
test 39m 56s e2e-live 81/81 incl. 2 new scenarios; static 60+21 skipped; unit 125/125
create-pr 1m 32s draft PR #741 + hickey+lowy analysis comment
ci 1m 54s vira ci both signoffs; e2e-morph 81/81
evidence 2m 22s three before/middle/after screenshots
Total 2h 19m 22s

Slowest step: test (39m 56s) — dominated by an e2e-live run on a binary built before the fsnotify root-cause was found.

Optimization suggestions

  • Verify the live happy path manually before running the full e2e suite. test burned ~25 minutes on a green-looking build that the new @hot-reload scenarios were silently failing against. A 60-second cp fixture → emanote run → edit .emanoteignore → curl would have surfaced the **/.*/** filter-pattern bug before the slow Cucumber + Playwright pass.
  • Pre-test filepattern semantics for any pattern you change. The whole second hour of this run was triggered by **/.*/** accidentally matching root-level dotfiles (filepattern's ** matches the empty path). A 3-line ?== smoke test would have caught this at design time, in the research step.
  • Combine related police fixes that touch the same file. The seven elegance commits each round-tripped through nix develop -c cabal build emanote + just fmt (~30s each). Batching the trivial in-file ones (e.g. lmlRouteFileType + trim-Haddock + collapse-TVars all touched Source/Dynamic.hs) would shave a few minutes off without losing the per-finding commit message clarity.
  • Re-use --from polish or --from ci-only on retries. The pattern fix needed only a rebuild + e2e replay, but it walked the full workflow from check onwards. The CLI already supports skipping to the relevant entry point.

Workflow completed at 2026-05-17.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hot-reload .emanoteignore mid-session

1 participant